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A meta-assessment is provided of 15 pilot projects 
aimed at demonstrating how student outcomes of various kinds can be 
used to assess the effectiveness of general education and 
baccalaureate degree programs. The Student Outcomes Assessment 
projects funded by Academic Program Improvement grants of the 
California state university system, have developed and field tested a 
variety of assessment measures and instruments, including portfolios, 
interviews, senior and capstone projects, surveys, and examinations. 
Factors that may have contributed to the outcomes were evaluated 
with relevant variables conceptualized within the categories of 
assessment environment, assessment methods, and assessment outcomes. 
Evidence for many of the direct and indirect outcomes proved 
difficult to evaluate, in part because some of the target outcomes 
are long-term effects that would require longitudinal measurement. 
The outcome that appeared to be the best index of project 
effectiveness was dissemination of results. Information sharing was 
consistently identified as an important outcome of project 
involvement. Human resources were the most important environmental 
variables, while the key methodological variable was development or 
adoption of adequate measures of outcomes. Six tables present 
information about the projects, and there is one figure of factors 
and outcomes. (SLD) 
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INTRODUCTION 

The current assessment movement in hig'her education is 
dri\'en by the wary partnership of reform and accountability, a 
partnership yielding a complex and diverse collection of 
asses??ment activities in university settings {Ewell, 1991). Over 
the last several years, the California State University System 
has moved to construct an assessment agenda which responds to 
both reform and accountability in ways that will preser\-e the 
commitment of the CSU to Intel lecti.al and programmatic diversity. 
Beginning in 1986, Academic Program Improvement grant funds ha\-e 
supported a series of student outcomes assessment projects 
initiated by faculty i.n a variety of disciplines on 11 campuses 
of the CSU. These pilot projects, under the aegis of the CSU 
Institute for Teaching and Learning, were aimed at demonstrating 
how student outcomes of various kinds can be used to. assess the 
effectiveness of General Education and baccalaureate degree 
programs . 

The present study, sponsored by the Fund for the Im]:>ro\-enienn 
of Postsecondary Education (FIPSE), provides a ''me ta-assessment" 
of the 15 pilot projects. These projects, briefly detailed in 
Table 1, have developed and field-tested a variety of assessment 
measures/ instruments , including port fol ios , intervi ews , 
senior/capstone projects, surveys, and examinations. 

This "meta-assessment" of the projects was des i gned to 
identify critical features across the fifteen pr'ojects which uere 
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consistently associated with effective assessment:. Once defined, 
these features might be expected to facilitate wider 
implementation of the assessment strategies f ield- tested 
experimentally. Because they represented a broad range of 
disciplinary perspectives, the pilot projects offered an 
opportunity to define factors which predicted assessment 
effectiveness across curricular boundaries. 

METHOD 

The project used a multi-site method to evaluate factors 
that may have contributed to the outcomes achieved. A three-part 
framework was used to specify variables and to organize the data. 
Relevant variables were conceptual i Z'l^d within the categories of 
assessment environment, assessment methods, and assessment 
outcomes . 

Assessment Envi ronment 

The assessment environment was defined as the social and 
organizational setting in which each project occurred. Variables 
relevant to the determination of this construct included : 

1) general faculty participation in the planning of 

the pro j ec t 

2) faculty participation in the implementation of the 

project 

3 ) the f acuity ' s perce ived *' ownership" of the pro jec t 

(i.e., self- de termination) 

4 ) faculty consensus with the project plan 

5 ) faculty workload required by the project 

6) the project director's exper i ence in assessment 

activities 

7) the faculty's experience in assessment activities 

8) the project director's general academic experience 

9) adequacy of budget, supplies and resources 

10) administrative support 

11) student support 

12) previous experience with outcomes assessment 
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13) the content domain (e.g., physical science) 

14) and the nature of the intended audience for the 

project ' s results 

Assessment Methods 

Assessment methods were defined by variables that described 
the strategic aspects of each project. General procedures with 
relevance across curriculums were selected. These variables 
inc I uded : 

15 ) goal definition 

16) selection of appropriate outcomes 

17) the psychometric adequacy of measures used or 

developed 

18) the success of data collection and reporting 

19) the appropriate use of statistical analyses 

20) sensitivity to multicultural issues 

21) the use/deyelopment of multiple outcome measures 

22) the comprehensiveness of reports describing the 

results of the project 

23) and the utility/economy of procedures used 

Assessment Outcomes 

Outcome variables were used to capture the systematic growth 
and change that could be attributed to the assessment project. 
The following "direct'' and "indirect*' outcomes of the assessment 
projects were evaluated : 

DIRECT 

24) the survival of the project 

25) attainment of additional funding 

26) gains in student achievement 

27) curricular development 

28 ) improved teaching 

29 ) better student feedback 

30) the use of new methods of assessment 

31) improved use of existing databases 

32) and the dissemination of results 

INDIRECT 

33 ) improved student recruitment/retention 

34^ better general attitudes toward assessment 
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35 ) new sources of money for assessment 

36 ) higher visibility of assessment 

37) and external adoption of measures or methods 

developed 



Conceptual Model 

The conceptual model guiding the analysis is based on the 
assumption that key "environmental" and "methodological" 
variables will determine the nature of "outcome" variables. This 
model is illustrated in Figure 1. Included in each of the 
categorical "bins" are the specific variables that were 
addressed . 
D ata Collection 

Information relevant to the assessment of the variables 
described above was obtained from multiple sources. First, 
copies of all project reports were obtained. All reports were 
studied and assessed by both researchers conducting this analy- 
sis. Independent conclusions concerning the "presence/absence" 
or "adequacy/ j nadequacy" of key variables were compared and 
evaluated. Where consensus on variable indicators could not be 
reached, that source of information was dropped from 
consideration. This process was especially important to the 
assessment of methodological variables (variables ^15 to ^^23). 

The second, and perhaps most utilized, source of data was 
the project directors themselves. A sur\'ey was developed and 
administered to each of the project directors. Results of this 
survey were especially critical in determining the nature of 
environmental variables for each project. These results were 
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also primary determinants of the project outcome variables. 

Project directors were further utilized throuijhout the data 
collection period to fill in missing information and to 
provide updates on developments not available in the final 
project reports. A meeting with directors held in the third 
month of the data collection process yielded additional 
information on the assessment environment at individual campuses. 

Telephone interviews with project d i rectors continued into 
the final weeks of the data collection process, A special effort 
was made to obtain additional focus concerning multicultural 
issues relevant to outcomes assessment. 

The final source of i nf ormat ion- ut i 1 ized was the reports 
submitted by the external evaluators of the pilot projects. 
These were read last in an effort to maintain the objectivity of 
the researchers through the initial stages of the data 
collection. These reports were especially useful in 
supplementing observations concerning the methodological 
variables . 
Analysi s 

Data organization was achieved in a manner consistent with 
the recommendations for qualitative data analysis prescribed by 
Miles and Huberman (1984). All observations, comments, and 
survey scores were condensed and entered in a highly abbreviated 
form onto a "me ta-matr ix . This master chart contained 
information relevant to all thirty-seven variables for all 
projects considered complete enough for inclusion in the final 
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assessment. In early versions of the chart, the basic principle 
was inclusion of all relevant data. 

For the final analysis, projects which lacked comprehensive 
data were excluded. One project assessed three different 
disciplines on three different campuses. Because the five 
sources providing informat ioti for this analysis were describing 
discrete event from what were apparently very different 
experiences, these data sources were treated as five different 
"sites'* for this analysis. The total "n" of sites ultimately 
used was sixteen. 

On the basis of information represented in this matrix, a 

qualitative categor-izat ion of all projects on all variables was 

completed. For each project, all variables were classified as: 

4 - strongly present/achieved , 
3 - partly pre sent /achieved , 
2 - weakly present /achieved , 
1 - absent/not achieved. 

The results of this classification are presented in Table 2. 

Projects are presented in random order. This display is referred 

to by Miles and Huberman (1984) as a "Site-Ordered Descriptive 

Meta-Matrix. " Though the "values" are the result of qualitative 

assessment and therefore include a certain unavoidable level of 

judgement, almost all are based on multiple sources of 

information. The agreement between sources was generally very 

compelling . 

Using the proposed research conceptualization, project 
outcomes (direct and indirect) would logically represent 
dependent measures. Environmental and methodological variables 
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would be the multiple predictors. If the data were interval, and 
the n of projects greater, the analysis of choice would be 
multiple regression. In a multiple regression paradigm, critical 
factors that should be attended to in any replication would be 
those that accounted for the most variance in the dependent 
measure (program outcomes). 

However, the data presented in Table 2 is qualitative, and 
the n of projects is sixteen. Nevertheless, the problem is 
conceptually the same. One must identify the variables that were 
consistently associated with project effectiveness. The task is 
to determine which predictor variables most consistently co-vary 
with outcomes. Good predictors are those whose qualitative 
categorization is most consistent with the qualitative 
categorization of project outcomes, 

Because a project's relative success should not be judged by 
a single outcome variable, and because there should be some 
resistance to the temptation to over-analyze qualitative data, 
outcomes were combined. Subsequent analyses were conducted 
using: 1) the average of all outcomes, 2) the average of outcomes 
categorized as direct outcomes, and 3) the average of indirect 
outcomes (refer back to page 6 for a precise listing of these 
outcomes ) . 

The next step necessary to enable comparison across 
variables is a standardization of scores. Though the "scale" 
used for classification of all predictor and outcome variables 
was the same (1 through 4), the mean and standard deviation of 
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che resulting distributions for each variable was different. 
This was resolved by transforming all scores to z-'scores. These 
scores represent the project's deviation above or below each 
given variable*3 mean in standard deviation units. 

*'Cons istency " be t ween predictors and outcomes was assessed 
by computing the squared deviations between each standardized 
score and the average standardized '^outcome" (overall, direct, & 
indirect) for the project associated with that score. 

Finally, these squared deviations were summed across project 
sites, and divided by the number of projects to produce a value 
conceptually similar to variance (the difference being that the 
source of deviation was between each predictor score and its 
project's outcome score rather than between each predictor score 
and the average of that predictor score across sites). The 
resulting ''variance with outcomes'* across sites will be smallest 
for those variables that were most consistently related to 
outcomes. Larger values indicate that the classifications 
assigned to that variable were not as predictive of the project 
outcomes . 

Results of this analysis are presented in Tables 3, 4, and 
5. Because of the limited nature of the data, there can be no 
assessment of statistical probability or significant differences. 
The predictor variables are simply ranked in the order of their 
consistency with outcomes: overall, direct, and indirect. Top- 
ranked variables are tVio se whose classifications were most 
consistent with project outcomes. 
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These results are summarized in Table 6, This table enables 
comparison of the relative importance of predictors across 
different combinations of project outcomes. Predictors are also 
organized into groups by similarity. 

RESULTS /DISCU SS ION 
ENVIRONMENTAL PREDICTORS 
Faculty Involvement Variables 

Two faculty variables were consistently toward the top of 
the respective rank orderings. "Faculty Participation" and 
"Perceived Faculty Workload" appear to be important indicators of 
assessment success. The first is most closely associated with 
those outcomes identified as "direct" outcomes. The second is 
more closely associated with "indirect" outcomes. "Faculty 
Ownership" and "Faculty Consensus with Plan" were also re La ivel y 
g'ood indicators, especially for direct outcomes. "Faculty 
Participation in Planning" appears to have little to do with 
project ou tcomes . 

It was expected that faculty involvement would be critical 
to project outcomes. Consequently, it was no surprise that 
faculty participation in the project was ranked number one in its 
association with the direct, intended outcomes of each project. 
Faculty consensus with the project plan, ownership of the 
project, and perceived workload were also ranked in the top ten 
in reference to direct outcomes. 

Perceived workload appeared especially critical to indirect 
outcomes. Perhaps as workload increases, it becomes less likely 
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that indirect outcomes will be realized. Workload would also 
seem especially salient to the development of attitudes t-.oward 
outcomes assessment (one of the indirect outcomes). 

Somewhat surpriHing was the apparent unimportance of 
involving faculty in the early stages of project planning*. In 
the general management literature, it is often suggested that 
participative planning of projects is a good strategy for 
insuring project participation. Apparently in these academic 
settings, it was only critical that you do an adequate job of 
"selling" your plan once it was developed. 
Trainin g /Experience with Student Outcomes Assessment 

Project outcomes were associated with the project director's 
efficacy in outcomes assessment, especially indirect outcomes. 
Faculty training and the project director's general academic 
background and experience were not as closely tied to project 
results . 

Of the three factors describing the training and background 
of project participants, only the project director's training and 
experience specific to student outcomes assessment ranked very 
high in association with outcomes. This factor was especially 
critical to the attainment of indirect outcomes. This result is 
logical given that the director was generally responsible for the 
adequacy of project goals and methods . 
Support Varia bles 

Administrative support was closely tied to project outcomes, 
both di rect and ind i rect . Adequacy of budget , suppl ies , and 
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other institutional resources was not closely tied to project 
results. Student support appears only moderately associated with 
direct outcomes . 

Nothing happens without administrative support . In ail 
hierarchies, policy tends to come from the top down. Though a 
goal of the CSU system is to establish faculty- initiated outcomes 
assessment, these initiatives will most likely fail without the 
support of academic administrators. These individuals are in the 
position to set the "tone" in reference to the academic 
legitimacy of such efforts. Unless outcomes assessment efforts 
are rewarded, it will be difficult to maintain motivation to 
assess student outcomes. 

Student support did not rank high. Though this source of 
support was assessed through the eyes of the project directors 
rather than from the students themselves, this result does not 
appear unreasonable . 

What does seem somewhat surprising is the apparent 
unimportance of adequate budget and supplies. Some insight into 
this result was obtained in discussions with project directors 
about how much of their project was coming *'out of their hide.'* 
The efforts of some directors clearly went beyond what might have 
been expected given the modest budgets they were receiving for 
their administration of each project. This phenomenon was more 
likely to occur if the director was working in a content area for 
which outcomes assessment research could be considered legitimate 
professional development or for senior faculty members who were 
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no longer struggling to achieve tenure or promotion. 

Another possible explanation for the inconsistency between 
the adequacy of budget/ suppl ies and project success would be the 
inherent differences in requirements for adequate assessment 
procedures across different content areas. It may be possible 
that adequate assessment can be achieved for relatively smal 1 
costs in some disciplines, but will tend to be very expensive in 
others. Consequently, adequate results might be obtained for 
some even when resources are tight while others will find 
assessment prohibitive without adequate budgetary support. 
Existing Student Outcomes Assessment Procedures 

Whether or not the project was initiated in an environment 
that currently included some type of formal outcomes assessment 
did not appear to have a critical impact on project outcomes. 
T'le good news of this result is that projects breaking new ground 
do not necessarily have to anticipate damaging levels of 
resistance. The bad news may be that previous experience with 
outcomes assessment apparently may not guarantee that new 
initiatives will be welcomed with open arms. 
Project Focus 

The content area in which the project occurred had little to 
do with the relative success of the project. However, the 
intended breadth of the audience (who and how many 
individuals/organization I expect to learn about the results of 
my project) was very closely associated with outcomes, especially 
indirect outcomes . 
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There was some expectation that perhaps the content area in 
which the project occurred might have considerable impact on the 
relative success of the project. This may disappoint some 
educational or behavioral science researchers who might have 
liked to assume they had a corner on this part of the research 
market. This is a good result for a system that hopes to 
initiate outcomes assessment across a universe of content 
domains . 

The tight association between the intended breadth of the 
audience for the project's results and project outcomes is 
somewhat surprising. In general, this association is logical in 
that the excellence- of project outcomes should be related to how 
many individuals or organizations with which the director intends 
to share the results. This result may also reflect the 
director's experience and enthusiasm for student outcomes 
assessment. As previously observed, project director efficacy in 
outcomes assessment was closely associated with project outcomes. 
METHODOLOGICAL PREDICTORS 
General Procedural Adequacy 

The single most critical variable from this category was the 
development of good measures of student outcomes. The creation 
of good measures was predictive of project results, direct and 
indirect. Appropriate selection of outcomes to measure was also 
closely associated with project. outcomes . 

"Project Goal Definition and the adequacy of "Statistical 
Analyses'* conducted were moderately tied to indirect outcomes. 
These variables along with "Data Collection/Reporting" and the 
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reporting of "Measurement Properties" did not appear to co-vary 
tightly with direct project outcomes. 

In reference to general procedural adequacj'^, the most 
important aspect of the assessment projects appeared to be the 
adequacy of the measures they developed or adopted. It is highly 
logical to expect that project success would hinge on the ability 
of the measures used to reTiably and validly measure student 
outcomes. This process begins with the selection of appropriate 
outcomes to measure. 

Other variables in this category, though not highly ranked 
in their association with direct outcomes, were toward the top of 
the list in relation to indirect outcomes. Much of this 
relationship appeared dependent upon the association with the 
indirect outcome of external adoption* Those projects bound for 
adoption were generally the most precise in the definition of 
project goals and most ambitious and accurate in the production 
of statistical analyses. 
Project Comprehensiveness 

The "Development of Multiple Measures" of student outcomes 
and "Sensitivity to Multicultural Issues" appeared moderately 
associated with project outcomes. The comurehensiveness of the 
reports made available were not closely related to project 
outcomes , 

The relatively high ranking of "sensitivity to multicultural 
issues" and the development of multiple measures may again 
reflect the sophistication of the project director in reference 
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to good outcomes assessment. It is also reasonable to expect 
projects that developed or used more than one form of assessment 
to be more successful . 

Sensitivity to multicultural issues would appear espec ially 
relevant to the potential for recruitment and retention of under- 
represented students. This would partially explain this 
variables relatively high ranking in reference to indirect 
outcomes. 

The nature of final project reports varied widely. The low 
association between the comprehensiveness of project reports and 
project outcomes might indicate that much occurred that was not 
completely reported. Many directors indicated that deadlines 
fell before they had time to adequately process the project 
results. Some compensated by disseminating results via other 
channels (regional and national presentations, journal 
publications). Though this project followed up and received many 
of these reports, it appears that much information from 
successful, coporehensive projects was lost to the CSU system due 
to the fact that final reporting was required before assessment 
procedures could be adequately evaluated . 
Cost Ef f ect iveness of Project 

The "Utility/Economy'* of the projects ' assess ..lent procedures 
was near the bottom of the rank ordering. Apparently expensive 
projects in terms of their dollars spent to students assessed 
were not always the richest in results. 

This result is a perfectly reasonable result given the 



Characteristics of Successful SOA - 17 
experimental nature of these projects. Directors "tried out" a 
wide variety of assessment procedures. There will also always be 
some cost differences across disciplines necessitated by the 
different nature of assessment processes that must occur. Some 
content areas may be able to conduct excellent "cheap" 
assessment, while other areas may only achieve moderate results 
despite a high price tag. 

Nevertheless, there is some logic to the assumption that 
efficient assessment will produce more desirable results. As 
more data is collected, this relationship should be reassessed 
within content areas. 

Though qualitat ive analysis does not enable tlie same level 
of precision as might be obtained from more quantitative data, 
the observations made in this study were systematic, relatively 
objective, and almost always based on multiple sources. 
Additional testing and replications of this study's conclusions 
is suggested, and could occur within the CSU's continuing program 
to develop and monitor programs of student outcomes assessment. 

MAJOR CONCLUSIONS 

The major conclusions drawn from an overview of the 
project's results are organized in terms of the conceptual 
model ' s environmental and methodological variables. 
Environmental Variables 

The environmental variables critical to project outcomes 
were human resources. First, recruiting and maintaining faculty 
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support was a key variable, but one which showed high variability 
across projects. The factors underlying this variability are 
difficult to detect in the quantitative measures, but they appear 
in the interview data from project directors. One concern in 
nearly all faculty groups is ''the intended primary use of the 
outcomes data," particularly where data suggest evidence of 
teaching/program effectiveness. Another might be described as 
the worry over the human capital costs of department-level 
assessment activities. This turned up as a particular concern 
for Junior faculty. Many junior faculty perceive that research 
on teaching and learning is regarded as ''second-tier" research 
which may not be counted in the tenure/promotion process. As 
several respondents indicated, assessment activities favor two 
faculty groups: 1) those in social /behavioral disciplines, and 2) 
those whose professional research activities "fit" with 
assessment research. 

Drawing from the experiences of the project directors, there 
appear to be several general guidelines for establishing and 
maintaining faculty participation: 

1 ) Educate participants about the value of assessing 
student outcomes , The motivation re qui red to commit to 
outcomes assessment is dependent upon a general 
perception that this effort will provide payoffs. The 
general descriptions of outcomes attained (refer to 
survey results) provide more than adequate examples of 
the bene f its achieved by ef f ect ive programs , On an 
individual basis, participation will be enhanced if 
assessment activities are an integral part of the 
faculty performance criteria, 

2) Maintain local control of the project. Support was 
never achieved in one project partly because the 
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program was perceived as " someone else ' s grand plan 
that got shoved down our throats Support for another 
project was in jeopardy when the local faculty senate 
returned the project's proposal as an external mandate, 

3) Overcome the "threat" associated by many faculty 
with assessing outcomes. Many directors reported the 
suspicion expressed by their faculty concerning the 
intended use of the data collected. Faculty often saw 
the potential for information obtained to be used as a 
club to punish programs or individuals rather than as a 
constructive tool for self -development • 

The importance of the development of administrative support 
was consistently reported across projects. Data from the 
majority of respondents suggested that "in-kind" resources, 
publicity, campus-level coordination, and establishing a climate 
receptive of assessment initiatives were important positive 
contributions of administrative offices. Even in this positive 
environment however, data indicate that more concrete evidence of 
integration of the assessment agenda in campus-level policy and 
in concrete recognition of assessment activities for professional 
development are needed. 

Two general prescriptions can be made on the basi s o f 

director responses. The first deals with the establishment of 

administrative support, the second partially defines the nature 

of the support sought: 

1 ) Educate administrators about the value of assessing 
student outcomes. Just as executives of business 
organizations value economic indicators for their 
companies , educational admini st rators need to 
understand the potential value of performance feedback 
inherent in the assessment of student outcomes , They 
must also be sensitized to the need for discretionary, 
constructive use of such data. The validity of 
departmental reports will quickly erode if messengers 
delivering "bad news" are shot. 
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2) Administrators should recognize and reward the 
development of outcomes assessment in one's field as 
legitimate professional development . Many directors 
reported that contriution of time to outcomes 
assessment was unrealistic for all but senior, tenured 
faculty who could afford to "waste some time," The 
term "waste" was used only in reference to the 
potential for recognition of these e f forts by chai rs , 
deans , and department/school/universi ty evaluation 
committees in control of the retention, promotion and 
tenure process. 

Finally, project directors' training/experience in 
measurement and analysis was key to project effectiveness, and 
here there were important differences. Some project directors 
reported dismay over the difficulties in learning assessment 
procc'dures as the project progressed, A number of respondents 
echoed the sentiments of one director who felt that the project 
lost momentum "just as experience and proficiency began to 
develop • " 

These observations clearly suggest the importance of 
training in outcomes assessment for those administrating 
assessment development. Two general prescriptions flow from the 
comments made by project leaders: 

1) Assess the training/experience of project directors 
specific to assessing student outcomes • General 
knowledge and experience as an educator is not enough. 
This report's "assessment" of relevant knowledge was 
dependent upon self-report. This approach appeared 
satisfactory, producing large variance in the levels of 
described outcomes assessment expertise. 

2) Provide continued opportunities for training in and 
exposure to outcomes assessment. Many of the directors 
reported that the system-wide conferences and reports 
on outcomes assessment had been responsible for 
kindling their interest in the process. If faculty- 
based outcomes assessment is to be "institutionalized" 
within the CSU, we must continue to share knowledge. 
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Methodolog ical Variables 

The key methodological variable was the development or 
adoption of adequate measures of student outcomes. Adequacy of 
measurement implied several aspects. The first is the simple 
psychometric properties of the assessment. This involves the 
reliability and validity of the assessment procedures. As an 
example of awareness of measurement adequacy, several directors 
did an excellent job of assessing the inter-rater reliability of 
judges producing qual itative assessments of student projects or 
papers. Others spent considerable time and consulted widely with 
their peers to evaluate the content validity of their 
assessments. This process often had positive, retroactive impact 
on curriculum and teaching strategies. 

Multicultural sensitivity also contributed to the adequacy 
of measurement, especially in reference to the inferences drawn 
froM assessment scores. A critical question for directors to ask 
was, "what assessment procedures will provide all students w^ith 
an equitable opportunity to demonstrate their competence?" This 
consideration should result in the production of multiple, more 
creative indices that x^ould provide a more comprehensive picture 
of student achievement. 

Multiple types of assessment also enabled directors to 
obtain feedback on more than one type of outcome. Rather than 
focusing solely on content or cogni t ively-based outcomes, 
additional measures of affective and attitudinal variables 
resulted in a much richer, more complex basis for judgements of 
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prog ram adequacy • 

The bottom line is that measures cannot be haphazardly 
developed or selected. Future faculty initiators of student 
outcomes assessment should not be told if, when, and what to 
assess, but most may benefit from technical support in reference 
to how to assess, 

The following summary prescriptions are derived from 
observations related to procedural and measurement adequacy: 

1) Clearly define educational objectives. The nature 
of the assessment tools cannot be determined until the 
desired outcomes are described. Several directors 
reported that healthy re-evaluations of their 
curriculum and program goals were a necessary precursor 
to the development of the actual assessment 
instruments. 

2) Assess multiple outcomes. Directors indicated that 
since educational objectives were seldom 
unidimensional , it made little sense to attempt to 
assess educational criteria with a single measure. The 
richest data sources enabling the clearest assessment 
of program outcomes involved combinations of content 
tests , performance-based demonstrations , attitude 
assessments, affective measures, etc. 

3) . Sensitivity to test fairness across constituent 
groups. Not all directors dealt with this issue, but 
those who did provided ample evidence of differential 
performance across groups. Though some differences may 
validly reflect the results of disadvantaged 
preparation for higher education, assessment procedures 
must minimize performance deficits related to native 
language diffe rences and/or content that contains 
cultural /socioeconomic bias. 

4) Assessment of measurement reliability/validity. 
The importance of this recommendation cannot be over- 
emphasized. If an assessment instrument does not 
possess adequate psychometric properties , it provides 
no basis for meaningful inference concerning the 
relative performance of the student or the success of 
the academic program. No amount of faculty and 
administrative support will save a project based on 
faulty measurement. 
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Outcomes 

The adequacy of project outcomes to provide direction for 
future assessment initiatives is a critical dimension of project 
effectiveness, Evidetice for many of the direct and indirect 
outcomes anticipated in the model for the study proved difficult 
to evaluate, in part because some of the target outcomes are 
long-term effects which would require longitudinal measurement. 
In addition, some data on outcome variables were still being 
collected as project final reports fell due, so did not get 
included as outcomes. 

The outcome which may be the best index of project 
effectiveness appeared to be dissemination of results. Project 
directors' verbal reports consistently underscored the importance 
of information sharing as an outcome of project involvement. 
Moreover, the projects with explicit descriptions of plans for 
dissemination to an identi f ied audience in their early goal 
definition continue to be active in publishing results. Further 
study of these projects and their results should provide 
additional direction for future strategies in f acul ty- initiated 
assessment programs . 
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'aale 2: Project Raw Scores cn All Vanaoles 



PROJECTS 



VARIABLES 



1 2 3 4 5 



I 8 9 10 11 i; 



PLAKHIH5 PARTICIP.4TI0N 1 2 2 

FAC PART IN PROJECT 2 f 2 

FACULTY OWNERSHIP 1 4 2 

CONSENSUS WITH PLAN 3 4 2 

PERCEIVED WORKLOAD 3 4 2 



4 2 2 
3 1 3 
2 3 3 



2 2 2 3 4 2 



DIRECTOR'S TRAINING 3 3 3 

FACULTY TRAINING 2 4 1 

DIRECTOR'S ACAD EXP 4 4 3 



BUDGET J SUPPLIES 
ADMINISTRATIVE SUPPCRT 
STUDENT SUPPORT 



4 1 



PREVIOUS ASSESSMENT 1 4 2 

HESH WITH EXISTIIiG PRGS 3 3 



1 1 



4 4 



CONTENT DOMAIN 
BREADTH OF AUDIENCE 



2 2 2 



2 1 



4 2 



GOAL DEFINITION 2 

SELECTION OF OUTCOMES 2 

MEASURES DEVELPED 2 

DATA COLLCTN/REPORTING 2 

HEASURHENT PROPERTIES 1 

STAT ANALYSIS ; 

HULTICULTURAL ISSUES 1 

DEVELOPMENT OF MULT HEAS 3 

REPORT COMPREHENSIVENESS 2 



4 3 

4 3 

2 2 

3 1 



4 4 



UTILITY/ECONOMY 



2 2 2 



2 2 



PROJECT CONTINUING 
ADDITIONAL FUNDING 
GAINS IN STUDENT ACH 
CURRICULAR IMPACT 
BETTER TEACHING 
FEEDBACK TO STUDENTS 
NEW MEASURES DEVELOPED 
BETTER DATA USE 
DISSEMINATION OF RESULTS 

RECRUITMENT/RETENTION 
ATTITUDES TOWARD ASSMNT 
MONEYS FOR ASSESSMENT 
VISIBILITY OF ASSESSMENT 
EXTERNAL ADOPTION 



1 



1 1 
4 1 

2 3 
4 3 
4 1 

3 1 
3 1 
1 3 



2 1 

2 3 

2 1 

4 3 

1 1 



1 i 
4 2 
4 3 
! 4 

3 3 

2 1 

1 ! 

4 2 

3 3 



1 



AVG DIRECT OUTCOMES 
'VG INDIRECT OUTCOMES 
0/ERALL AVG OUTCOME 



2.11 2.44 1.56 1.56 2.56 1.61 3.44 2.55 3.00 3.25 2.69 2.5? 2.c5 3.33 l.CO 3.17 
1.00 2.40 1.60 1.00 1.20 2.00 2.40 2.20 1.80 3.00 2.50 2.20 3.51 2.:0 l.C. 2. CO 
1.56 2.42 1.58 1.28 1.83 1.83 2.92 2.38 2.40 3.13 2.J4 2.43 3.23 2.5? 1.00 2.53 



ERIC 



30 



Characteristics of Successful SOA - 



Table 3: Predictor Variables Rank Ordered by Consistency with 
Overall Outcomes 



Var iance with 

Predictor Variables Rank Order Overall Outcomes 

Adequacy of Measures Dev. 1 .73 

Intended Breadth of Aud. 2 .75 

Administrative Support 3 .76 

Faculty Part, in Project 4 .77 

Director's Training in SOA 5 .91 

Perceived Faculty Workload 6 .95 

Sensitivity to Mult Issues 7 1.06 

Development of Mult Measures 8 1.10 

Selection of Outcomes 9 1.12 

Measurement Properties 10 1.22 

Statistical Analysis H 1.25 

Faculty Consensus with Plan 12 1.27 

Project Goal Definition 13 1.28 

Faculty Ownership of Project 14 1.31 

Data Collection/Reporting 15 1.39 

Existence of Previous SOA 16 1.50 

7aculty Training in SOA 17 1.60 

Student Support of Project IS 1.60 

Adequate Budge t / Supp 1 ies 19 1.71 

Content Domain 20 1.79 

Report Comprehensiveness 21 1.80 

Utility/Economy of Project 22 2.32 

Director's Academic Exper. 23 2.41 

Fac Participation in Planning 24 2.43 
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Table 4: Predictor Variables Rank Ordered by Consistency with Direct 
Outcomes 



Variance with 

Predictor Vari ables Rank Order Direct Outcomes 



Faculty Part, in Project 1 .56 

Adequacy of Measures Dev. 2 .80 

Administrative Support 3 .89 

Faculty Consensus with Plan 4 1.00 

Intended Breadth of Aud. 5 1.09 

Development of Mult Measures 6 1.09 

Faculty Ownership of Project 7 1.23 

Selection of Outcomes 8 1.23 

Perceived Faculty Workload 9 1.25 

Sensitivity to Mult Issues 10 1.27 

Student Support of Project 11 1.31 

Director's Training in SOA 12 1.32 

Measurement Properties 13 1.37 

Statistical Analysis 14 1.48 

Project Goal Definition 15 1.54 

Faculty Training in SOA 16 1.63 

Data Collection/Reporting 17 1.64 

Content Domain 18 1.83 

Existence of Previous SOA 19 1.85 

Adequate Budget/Supplies 20 1.86 

Utility/Economy of Project 21 2.07 

Report Comprehensiveness 22 2.38 

Director's Academic Exper. 23 2.41 

Fac Participation in Planning 24 2.45 
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Table 5: Predictor Variables Rank Ordered by Consistency with 
Indirect Outcomes 



Variance with 

Predictor Variables Rank Order Indirect Outcomes 

Intended Breadth of Aud. 1 .66 

Director's Training in SOA 2 .72 

Perceived Faculty Workload 3 .86 

Administx^ative Support 4 .89 

Adequacy of Measures Dev. 5 .92 

Sensitivity to Mult Issues 6 1.06 

Project Goal Definition 7 1.17 

Selection of Outcomes 8'" 1.17 

Statistical Analysis 9 1.20 

Faculty Part, in Project 10 1.22 

Measurement Properties 11 1.24 

Existence of Previous SOA 12 1.25 

Development of Mult Measures 13 1.29 

Data Collection/Reporting 14 1.36 

Report Comprehensiveness 15 1..44 

Faculty Ownership of Project 16 - 1.53 

Adequate Budget /Suppl ies 17 1.62 

Faculty Training in SOA 18 1.66 

Faculty Consensus with Plan 19 1.69 

Content Domain 20 1.80 

Student Support of Project 21 1.96 

Fac Participation in Planning 22 2.32 

Director's Academic Exper. 23 2.33 

Utility/Economy of Project 24 2.49 
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Table 6: Variance between Predictors and Outcomes with Predictor 
Variable Rank Orders* 



Varnce with Varnce with V'arnce with 
P redictor Variables All Outcomes Dir Outcomes Ind Outcomes 



Fac Part in Planning 


2 


43 


( 24 ) 


2 


4 5 


( 24 ) 


2 


32 


( 22 


Faculty Part, in Project 


0 


. ( 7 


( 4 ) 


0 


, 56 


( 1 ) 


1 


. 22 


( 10 


Fac Ownership of Project 


1 


31 


( 14 ) 


1 


.23 


( 8) 


1 


53 


( 16 


Fac Consensus w^ith Plan 


1 


.27 


( 12 ) 


1 


.00 


( 4) 


1 


. 69 


(19 


Perceived Faculty Workload 


0 


, 95 


( b ) 


< 
1 


2o 


( 9 ) 


0 


, 86 


( 3 


uirecLor s i raining in bUA 


0 


, y 1 


( 5 ) 


1 


, 32 


(12) 


0 


. 72 


( 2 


racuiu>r I raining in bUA 


1 


A 

bO 


( 1 M 


1 


63 


( 16 ) 


1 


66 


( 18 


Director's Academic Exper» 




, 4 1 


( i3 ) 


2 


, 4 1 


{ 23 ) 


0 


. 33 


( 23 


Adequate Budget /Supplies 


1 


71 


( 19 ) 


1 


86 


( 20) 


1 , 


62 


( 1 7 


.HumiriisuraLi\e oupporu 


A 
U 


7 C 
. ( 0 


( o ) 


0 


89 


( 3 ) 


0 


. 89 


( 4 


ouuutriiu onj[j}jox L oi r^rojecL 


1 

1 


R n 




1 . 


o 1 


/11V 

(11) 


1 , 


96 


( 2 1 


Existence of Previous SOA 


1 


50 


( 16 ) 


1 


85 


( 19 ) 


1. 


25 


( 12 


Content Domain 


1 


79 


( 20 ) 


1 


83 


( 18 ) 


1 


80 


( 20 


Intended Breadth of Aud. 


0 


75 


( 2) 


1 


09 


( 5) 


0 


, 66 


( 1 


Project Goal Definition 


1 


28 


( 13 ) 


1 


54 


( 15 ) 


■ 1 


17 


( 8 


Selection of Outcomes 


1 


12 


( 9 ) 


1 


23 


( 7 ) 


1 


17 


( 7 


Adequacy of Measures Dev. 


0 


73 


( 1 ) 


0, 


80 


( 2) 


0 


92 


( 5 


Data Col lect ion /Report ing 


1 


39 


( 15 ) 


1 


64 


( 17 ) 


1 


36 


( 14 


Measuremen t P roper t ies 


1 


22 


( 10 ) 


1 


37 


( 13 ) 


1 , 


24 


( 1 1 


Statistical Analysis 


1 


25 


( 11 ) 


1 


48 


( 14 ) 


1 


,20 


( 9 


Sensitivity to Mult Issues 


1 


06 


( 7 ) 


1 


27 


( 10 ) 


1 , 


06 


( 3 


Develpmnt of Mult Measures 


1 


10 


( 8) 


1 


09 


( 6) 


1 


.29 


( 13 


Report Comprehensiveness 


1 


80 


(21 ) 


2 


38 


( 22 ) 


1 . 


44 


( 15 


Utility/Economy of Project 


2 


32 


(22) 


2 


07 


(21 ) 


2 , 


49 


(24 
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FIGURE 1: Conceptual Model 
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